Stratified Sampling for Extreme Multi-label Data

Extreme multi-label classification (XML) is becoming increasingly relevant in the era of big data. Yet, there no method for effectively generating stratified partitions XML datasets. Instead, researchers typically rely on provided test-train splits that, 1) aren’t always representative entire dataset, and 2) are missing many labels. This can lead to poor generalization ability unreliable performance estimates, as has been established binary multi-class settings. As such, this paper presents a new simple algorithm that efficiently generate datasets with millions unique We also examine label distributions prevailing benchmark splits, investigate issues arise from using unrepresentative subsets data model development. The results highlight difficulty stratifying data, demonstrate importance training evaluation.